Interactive Visualisation and Dashboards
Contents
Interactive Visualisation and Dashboards#
import pypsa
import atlite
import pandas as pd
import geopandas as gpd
import xarray as xr
import matplotlib.pyplot as plt
from urllib.request import urlretrieve
plt.style.use('bmh')
fn = "network-cem.nc"
url = "https://tubcloud.tu-berlin.de/s/kpWaraGc9LeaxLK/download/" + fn
urlretrieve(url, fn)
('network-cem.nc', <http.client.HTTPMessage at 0x7fecc52ba730>)
Load Example Data#
First, let’s load a few example datasets you know from previous tutorials.
A PyPSA network:
n = pypsa.Network("network-cem.nc")
WARNING:pypsa.io:Importing network from PyPSA version v0.21.3 while current version is v0.22.1. Read the release notes at https://pypsa.readthedocs.io/en/latest/release_notes.html to prepare your network for import.
INFO:pypsa.io:Imported network network-cem.nc has buses, carriers, generators, global_constraints, loads, storage_units
n.lopf(solver_name='cbc')
INFO:pypsa.linopf:Prepare linear problem
INFO:pypsa.linopf:Total preparation time: 0.36s
INFO:pypsa.linopf:Solve linear problem using Cbc solver
INFO:pypsa.linopf:Optimization successful. Objective value: 6.58e+10
('ok', 'optimal')
Wind, solar and demand time series:
url = "https://raw.githubusercontent.com/fneum/data-science-for-esm/main/data-science-for-esm/time-series-lecture-2.csv"
ts = pd.read_csv(url, index_col=0, parse_dates=True)
Power plants in Europe
url = "https://raw.githubusercontent.com/PyPSA/powerplantmatching/master/powerplants.csv"
ppl = pd.read_csv(url, index_col=0)
geometry = gpd.points_from_xy(ppl['lon'], ppl['lat'])
ppl = gpd.GeoDataFrame(ppl, geometry=geometry, crs=4326)
NUTS2 regions:
url = "https://tubcloud.tu-berlin.de/s/RHZJrN8Dnfn26nr/download/NUTS_RG_10M_2021_4326.geojson"
nuts = gpd.read_file(url).set_index('id').query("LEVL_CODE == 2")
An atlite cutout:
cutout = atlite.Cutout("PORTUGAL-2013-01-era5.nc")
Limitations of Static Plotting with Matplotlib#
You will agree that using matplotlib for static plotting is great for reports, but that it’s lacking some features for interactive visualisation.
ts.onwind.plot(figsize=(10,2))
<AxesSubplot:>
There are many Python-based interactive plotting libraries out there, and it can be confusing to keep an overview. This tutorial introduces you to two of them:
hvPlot, which is a high-level API mostly for bokeh plots that integrates nicely with
pandas.plotly.express, which is a high-level API for plotly plots.
These two tools allow you to produce shiny interactive figures with minimal code, however, at the expense of fewer customisation options.
hvPlot#
.hvplot() is a powerful and interactive Pandas-like .plot() API. You just replace .plot() with .hvplot() and you get an interactive figure. Simple as that.
It can be installed via conda or mamba in the following way:
conda install -c pyviz hvplot geoviews
Documentation can be found here: https://hvplot.holoviz.org/index.html
To use it, we have to import hvplot.pandas, which makes the .hvplot accessor available on Pandas DataFrame and Series objects, which means that after that df.hvplot becomes a valid statement while before that it would raise an error.
import hvplot.pandas
Let’s try it by plotting onshore wind time series for the year…
ts.onwind.hvplot(height=200)
… or the load time series for February
ts.loc["2015-02", "load"].hvplot(height=200)
We can also plot geographic data with hvPlot, for instance, the locations of all hard coal power plants in Europe.
The geo=True declares that the data will be plotted in a geographic coordinate system.
Once hvPlot knows that your data is in geo-coordinates, you can use the tiles keyword argument to overlay a the plot on top of map tiles.
Note
For a list of available tiles, look here.
ppl.query("Fueltype == 'Hard Coal'").hvplot(
geo=True,
tiles=True,
frame_height=600
)
/home/runner/micromamba-root/envs/esm/lib/python3.9/site-packages/cartopy/crs.py:825: ShapelyDeprecationWarning:
__len__ for multi-part geometries is deprecated and will be removed in Shapely 2.0. Check the length of the `geoms` property instead to get the number of parts of a multi-part geometry.
/home/runner/micromamba-root/envs/esm/lib/python3.9/site-packages/cartopy/crs.py:877: ShapelyDeprecationWarning:
Iteration over multi-part geometries is deprecated and will be removed in Shapely 2.0. Use the `geoms` property to access the constituent parts of a multi-part geometry.
/home/runner/micromamba-root/envs/esm/lib/python3.9/site-packages/cartopy/crs.py:944: ShapelyDeprecationWarning:
__len__ for multi-part geometries is deprecated and will be removed in Shapely 2.0. Check the length of the `geoms` property instead to get the number of parts of a multi-part geometry.
/home/runner/micromamba-root/envs/esm/lib/python3.9/site-packages/cartopy/crs.py:825: ShapelyDeprecationWarning:
__len__ for multi-part geometries is deprecated and will be removed in Shapely 2.0. Check the length of the `geoms` property instead to get the number of parts of a multi-part geometry.
/home/runner/micromamba-root/envs/esm/lib/python3.9/site-packages/cartopy/crs.py:877: ShapelyDeprecationWarning:
Iteration over multi-part geometries is deprecated and will be removed in Shapely 2.0. Use the `geoms` property to access the constituent parts of a multi-part geometry.
/home/runner/micromamba-root/envs/esm/lib/python3.9/site-packages/cartopy/crs.py:944: ShapelyDeprecationWarning:
__len__ for multi-part geometries is deprecated and will be removed in Shapely 2.0. Check the length of the `geoms` property instead to get the number of parts of a multi-part geometry.
/home/runner/micromamba-root/envs/esm/lib/python3.9/site-packages/cartopy/crs.py:825: ShapelyDeprecationWarning:
__len__ for multi-part geometries is deprecated and will be removed in Shapely 2.0. Check the length of the `geoms` property instead to get the number of parts of a multi-part geometry.
/home/runner/micromamba-root/envs/esm/lib/python3.9/site-packages/cartopy/crs.py:877: ShapelyDeprecationWarning:
Iteration over multi-part geometries is deprecated and will be removed in Shapely 2.0. Use the `geoms` property to access the constituent parts of a multi-part geometry.
/home/runner/micromamba-root/envs/esm/lib/python3.9/site-packages/cartopy/crs.py:944: ShapelyDeprecationWarning:
__len__ for multi-part geometries is deprecated and will be removed in Shapely 2.0. Check the length of the `geoms` property instead to get the number of parts of a multi-part geometry.
Like in geopandas, we can tell hvPlot to plot the point sizes and colors according to columns of the pandas.DataFrame. We can also change the opacity with alpha and the colormap with cmap.
plot = ppl.query("Fueltype == 'Hard Coal'").hvplot(
geo=True,
tiles='CartoLight',
frame_height=600,
c='DateIn',
cmap='viridis',
s='Capacity',
alpha=0.6
)
plot
/home/runner/micromamba-root/envs/esm/lib/python3.9/site-packages/cartopy/crs.py:825: ShapelyDeprecationWarning:
__len__ for multi-part geometries is deprecated and will be removed in Shapely 2.0. Check the length of the `geoms` property instead to get the number of parts of a multi-part geometry.
/home/runner/micromamba-root/envs/esm/lib/python3.9/site-packages/cartopy/crs.py:877: ShapelyDeprecationWarning:
Iteration over multi-part geometries is deprecated and will be removed in Shapely 2.0. Use the `geoms` property to access the constituent parts of a multi-part geometry.
/home/runner/micromamba-root/envs/esm/lib/python3.9/site-packages/cartopy/crs.py:944: ShapelyDeprecationWarning:
__len__ for multi-part geometries is deprecated and will be removed in Shapely 2.0. Check the length of the `geoms` property instead to get the number of parts of a multi-part geometry.
/home/runner/micromamba-root/envs/esm/lib/python3.9/site-packages/cartopy/crs.py:825: ShapelyDeprecationWarning:
__len__ for multi-part geometries is deprecated and will be removed in Shapely 2.0. Check the length of the `geoms` property instead to get the number of parts of a multi-part geometry.
/home/runner/micromamba-root/envs/esm/lib/python3.9/site-packages/cartopy/crs.py:877: ShapelyDeprecationWarning:
Iteration over multi-part geometries is deprecated and will be removed in Shapely 2.0. Use the `geoms` property to access the constituent parts of a multi-part geometry.
/home/runner/micromamba-root/envs/esm/lib/python3.9/site-packages/cartopy/crs.py:944: ShapelyDeprecationWarning:
__len__ for multi-part geometries is deprecated and will be removed in Shapely 2.0. Check the length of the `geoms` property instead to get the number of parts of a multi-part geometry.
/home/runner/micromamba-root/envs/esm/lib/python3.9/site-packages/cartopy/crs.py:825: ShapelyDeprecationWarning:
__len__ for multi-part geometries is deprecated and will be removed in Shapely 2.0. Check the length of the `geoms` property instead to get the number of parts of a multi-part geometry.
/home/runner/micromamba-root/envs/esm/lib/python3.9/site-packages/cartopy/crs.py:877: ShapelyDeprecationWarning:
Iteration over multi-part geometries is deprecated and will be removed in Shapely 2.0. Use the `geoms` property to access the constituent parts of a multi-part geometry.
/home/runner/micromamba-root/envs/esm/lib/python3.9/site-packages/cartopy/crs.py:944: ShapelyDeprecationWarning:
__len__ for multi-part geometries is deprecated and will be removed in Shapely 2.0. Check the length of the `geoms` property instead to get the number of parts of a multi-part geometry.
There are a few more options of the graph we can tweak in the opts() section, like which tools should be activated by default.
plot = plot.opts(
xaxis=None,
yaxis=None,
active_tools=['pan', 'wheel_zoom']
)
plot
/home/runner/micromamba-root/envs/esm/lib/python3.9/site-packages/cartopy/crs.py:825: ShapelyDeprecationWarning:
__len__ for multi-part geometries is deprecated and will be removed in Shapely 2.0. Check the length of the `geoms` property instead to get the number of parts of a multi-part geometry.
/home/runner/micromamba-root/envs/esm/lib/python3.9/site-packages/cartopy/crs.py:877: ShapelyDeprecationWarning:
Iteration over multi-part geometries is deprecated and will be removed in Shapely 2.0. Use the `geoms` property to access the constituent parts of a multi-part geometry.
/home/runner/micromamba-root/envs/esm/lib/python3.9/site-packages/cartopy/crs.py:944: ShapelyDeprecationWarning:
__len__ for multi-part geometries is deprecated and will be removed in Shapely 2.0. Check the length of the `geoms` property instead to get the number of parts of a multi-part geometry.
/home/runner/micromamba-root/envs/esm/lib/python3.9/site-packages/cartopy/crs.py:825: ShapelyDeprecationWarning:
__len__ for multi-part geometries is deprecated and will be removed in Shapely 2.0. Check the length of the `geoms` property instead to get the number of parts of a multi-part geometry.
/home/runner/micromamba-root/envs/esm/lib/python3.9/site-packages/cartopy/crs.py:877: ShapelyDeprecationWarning:
Iteration over multi-part geometries is deprecated and will be removed in Shapely 2.0. Use the `geoms` property to access the constituent parts of a multi-part geometry.
/home/runner/micromamba-root/envs/esm/lib/python3.9/site-packages/cartopy/crs.py:944: ShapelyDeprecationWarning:
__len__ for multi-part geometries is deprecated and will be removed in Shapely 2.0. Check the length of the `geoms` property instead to get the number of parts of a multi-part geometry.
All this does not only work with points but also shapes. We can also pick the columns that should be shown when hovering on a shape using hover_cols.
nuts.hvplot(
geo=True,
tiles='OSM',
hover_cols=['NUTS_NAME', "NUTS_ID"],
c='CNTR_CODE',
frame_height=500,
alpha=0.2
).opts(
xaxis=None,
yaxis=None,
active_tools=['pan', 'wheel_zoom']
)
/home/runner/micromamba-root/envs/esm/lib/python3.9/site-packages/cartopy/crs.py:825: ShapelyDeprecationWarning:
__len__ for multi-part geometries is deprecated and will be removed in Shapely 2.0. Check the length of the `geoms` property instead to get the number of parts of a multi-part geometry.
/home/runner/micromamba-root/envs/esm/lib/python3.9/site-packages/cartopy/crs.py:877: ShapelyDeprecationWarning:
Iteration over multi-part geometries is deprecated and will be removed in Shapely 2.0. Use the `geoms` property to access the constituent parts of a multi-part geometry.
/home/runner/micromamba-root/envs/esm/lib/python3.9/site-packages/cartopy/crs.py:944: ShapelyDeprecationWarning:
__len__ for multi-part geometries is deprecated and will be removed in Shapely 2.0. Check the length of the `geoms` property instead to get the number of parts of a multi-part geometry.
/home/runner/micromamba-root/envs/esm/lib/python3.9/site-packages/cartopy/crs.py:825: ShapelyDeprecationWarning:
__len__ for multi-part geometries is deprecated and will be removed in Shapely 2.0. Check the length of the `geoms` property instead to get the number of parts of a multi-part geometry.
/home/runner/micromamba-root/envs/esm/lib/python3.9/site-packages/cartopy/crs.py:877: ShapelyDeprecationWarning:
Iteration over multi-part geometries is deprecated and will be removed in Shapely 2.0. Use the `geoms` property to access the constituent parts of a multi-part geometry.
/home/runner/micromamba-root/envs/esm/lib/python3.9/site-packages/cartopy/crs.py:944: ShapelyDeprecationWarning:
__len__ for multi-part geometries is deprecated and will be removed in Shapely 2.0. Check the length of the `geoms` property instead to get the number of parts of a multi-part geometry.
/home/runner/micromamba-root/envs/esm/lib/python3.9/site-packages/cartopy/crs.py:825: ShapelyDeprecationWarning:
__len__ for multi-part geometries is deprecated and will be removed in Shapely 2.0. Check the length of the `geoms` property instead to get the number of parts of a multi-part geometry.
/home/runner/micromamba-root/envs/esm/lib/python3.9/site-packages/cartopy/crs.py:877: ShapelyDeprecationWarning:
Iteration over multi-part geometries is deprecated and will be removed in Shapely 2.0. Use the `geoms` property to access the constituent parts of a multi-part geometry.
/home/runner/micromamba-root/envs/esm/lib/python3.9/site-packages/cartopy/crs.py:944: ShapelyDeprecationWarning:
__len__ for multi-part geometries is deprecated and will be removed in Shapely 2.0. Check the length of the `geoms` property instead to get the number of parts of a multi-part geometry.
/home/runner/micromamba-root/envs/esm/lib/python3.9/site-packages/cartopy/crs.py:825: ShapelyDeprecationWarning:
__len__ for multi-part geometries is deprecated and will be removed in Shapely 2.0. Check the length of the `geoms` property instead to get the number of parts of a multi-part geometry.
/home/runner/micromamba-root/envs/esm/lib/python3.9/site-packages/cartopy/crs.py:877: ShapelyDeprecationWarning:
Iteration over multi-part geometries is deprecated and will be removed in Shapely 2.0. Use the `geoms` property to access the constituent parts of a multi-part geometry.
/home/runner/micromamba-root/envs/esm/lib/python3.9/site-packages/cartopy/crs.py:944: ShapelyDeprecationWarning:
__len__ for multi-part geometries is deprecated and will be removed in Shapely 2.0. Check the length of the `geoms` property instead to get the number of parts of a multi-part geometry.
We can also use hvPlot for xarray datasets (e.g. atlite cutouts).
For that, we need to import the corresponding xarray accessors.
import hvplot.xarray
So let’s try it by plotting the wind speeds in Portugal as provided by ERA5. The nice thing you will notice is that it will automatically open a panel for dimensions that we did not select explicitly. In this case we can easily sweep across the time dimension. Notice also the customisation options we use here.
cutout.data.hvplot.quadmesh(
'x', 'y', 'wnd100m',
frame_height=500,
cmap='Blues',
geo=True,
tiles='CartoLight',
alpha=0.8,
padding=.5,
clim=(0,10)
)
/home/runner/micromamba-root/envs/esm/lib/python3.9/site-packages/cartopy/crs.py:825: ShapelyDeprecationWarning:
__len__ for multi-part geometries is deprecated and will be removed in Shapely 2.0. Check the length of the `geoms` property instead to get the number of parts of a multi-part geometry.
/home/runner/micromamba-root/envs/esm/lib/python3.9/site-packages/cartopy/crs.py:877: ShapelyDeprecationWarning:
Iteration over multi-part geometries is deprecated and will be removed in Shapely 2.0. Use the `geoms` property to access the constituent parts of a multi-part geometry.
/home/runner/micromamba-root/envs/esm/lib/python3.9/site-packages/cartopy/crs.py:944: ShapelyDeprecationWarning:
__len__ for multi-part geometries is deprecated and will be removed in Shapely 2.0. Check the length of the `geoms` property instead to get the number of parts of a multi-part geometry.
/home/runner/micromamba-root/envs/esm/lib/python3.9/site-packages/cartopy/crs.py:825: ShapelyDeprecationWarning:
__len__ for multi-part geometries is deprecated and will be removed in Shapely 2.0. Check the length of the `geoms` property instead to get the number of parts of a multi-part geometry.
/home/runner/micromamba-root/envs/esm/lib/python3.9/site-packages/cartopy/crs.py:877: ShapelyDeprecationWarning:
Iteration over multi-part geometries is deprecated and will be removed in Shapely 2.0. Use the `geoms` property to access the constituent parts of a multi-part geometry.
/home/runner/micromamba-root/envs/esm/lib/python3.9/site-packages/cartopy/crs.py:944: ShapelyDeprecationWarning:
__len__ for multi-part geometries is deprecated and will be removed in Shapely 2.0. Check the length of the `geoms` property instead to get the number of parts of a multi-part geometry.
/home/runner/micromamba-root/envs/esm/lib/python3.9/site-packages/cartopy/crs.py:825: ShapelyDeprecationWarning:
__len__ for multi-part geometries is deprecated and will be removed in Shapely 2.0. Check the length of the `geoms` property instead to get the number of parts of a multi-part geometry.
/home/runner/micromamba-root/envs/esm/lib/python3.9/site-packages/cartopy/crs.py:877: ShapelyDeprecationWarning:
Iteration over multi-part geometries is deprecated and will be removed in Shapely 2.0. Use the `geoms` property to access the constituent parts of a multi-part geometry.
/home/runner/micromamba-root/envs/esm/lib/python3.9/site-packages/cartopy/crs.py:944: ShapelyDeprecationWarning:
__len__ for multi-part geometries is deprecated and will be removed in Shapely 2.0. Check the length of the `geoms` property instead to get the number of parts of a multi-part geometry.
/home/runner/micromamba-root/envs/esm/lib/python3.9/site-packages/cartopy/crs.py:825: ShapelyDeprecationWarning:
__len__ for multi-part geometries is deprecated and will be removed in Shapely 2.0. Check the length of the `geoms` property instead to get the number of parts of a multi-part geometry.
/home/runner/micromamba-root/envs/esm/lib/python3.9/site-packages/cartopy/crs.py:877: ShapelyDeprecationWarning:
Iteration over multi-part geometries is deprecated and will be removed in Shapely 2.0. Use the `geoms` property to access the constituent parts of a multi-part geometry.
/home/runner/micromamba-root/envs/esm/lib/python3.9/site-packages/cartopy/crs.py:944: ShapelyDeprecationWarning:
__len__ for multi-part geometries is deprecated and will be removed in Shapely 2.0. Check the length of the `geoms` property instead to get the number of parts of a multi-part geometry.
We can also plot the time series of solar generation in Germany on a heatmap:
ts.hvplot.heatmap(x='index.hour', y='index.week', C='solar', cmap='blues').aggregate(function='mean')
/home/runner/micromamba-root/envs/esm/lib/python3.9/site-packages/hvplot/util.py:476: FutureWarning:
weekofyear and week have been deprecated, please use DatetimeIndex.isocalendar().week instead, which returns a Series. To exactly reproduce the behavior of week and weekofyear and return an Index, you may call pd.Int64Index(idx.isocalendar().week)
hvPlot also offers stacked area charts that come in handy for plotting the power dispatch of a solved PyPSA network:
dispatch = pd.concat([n.generators_t.p, n.storage_units_t.p], axis=1).loc["2015-02"].div(1e3)
dispatch.where(dispatch > 0, 0).hvplot.area(
stacked=True, line_width=0,
width=1300,
height=350,
hover=False,
color=[n.carriers.at[c, 'color'] for c in dispatch.columns],
ylabel='electricity supply [GW]',
ylim=(0,180)
)
hvPlot also has a nice explorer that can be displayed in a Jupyter notebook and that can be used to quickly create customized plots.
hvplot.explorer(pd.DataFrame(ppl))
Plotly Express#
The
plotly.expressmodule (usually imported as px) contains functions that can create entire figures at once. Plotly Express is a built-in part of theplotlylibrary, and is the recommended starting point for creating most common figures. Every Plotly Express function uses graph objects internally and returns a plotly.graph_objects.Figure instance. Throughout the plotly documentation, you will find the Plotly Express way of building figures at the top of any applicable page, followed by a section on how to use graph objects to build similar figures. Any figure created in a single function call with Plotly Express could be created using graph objects alone, but with between 5 and 100 times more code.
Documentation is available here: https://plotly.com/python/plotly-express/
It can be installed via conda or mamba in the following way:
conda install -c conda-forge plotly
import plotly.io as pio
import plotly.express as px
import plotly.offline as py
Note
We need to import plotly.io and plotly.offline, so that the interactive plots are also visible on the course’s static website.
Let’s reproduce the plots we previously created with hvPlot. Onshore wind capacity factor time series:
px.line(ts.onwind)
Load time series in February:
px.line(ts.loc["2015-02", "load"])
Hard coal power plants in Europe:
df = ppl.query("Fueltype == 'Hard Coal'")
px.scatter_mapbox(
df,
lat="lat",
lon="lon",
mapbox_style="carto-positron",
zoom=2,
height=600
)
px.scatter_mapbox(
df,
lat="lat",
lon="lon",
mapbox_style="carto-positron",
color="DateIn",
size="Capacity",
zoom=2,
height=600
)
px.choropleth_mapbox(
nuts,
geojson=nuts.geometry,
locations=nuts.index,
mapbox_style="carto-positron",
zoom=2,
height=600,
color="CNTR_CODE",
center={"lat": 48, "lon": 12}
)